How can an Ethereum contract get data from a website?

Question

What is the process/workflow behind the process where an Ethereum contract gets some data from a website?

Answer 1

You can use an Oracle. An oracle is any device or entity that connects real-world data with the blockchain.

There are several examples of oracle technologies. Chainlink and Provable (formerly Oraclize) are two examples that do exactly that.

There are some code examples here as well as the documentation of our Solidity API.

Oraclize is available both on mainnet and testnet so poking around with it should be easy, however if you need any support feel free to ask - we even have a gitter channel here.

As you can see fetching data from a website is as easy as using the oraclize_query function.

Our good behaviour is granted by the TLSNotary proof and can be easily checked with this web based client-side network monitor.

For example to fetch the ETHXBT price from the Kraken ticker:

import "dev.oraclize.it/api.sol";

contract KrakenPriceTicker is usingOraclize {
    string public ETHXBT;

    function PriceTicker() {
        oraclize_setNetwork(networkID_testnet);
        oraclize_setProof(proofType_TLSNotary | proofStorage_IPFS);
        oraclize_query("URL", "json(https://api.kraken.com/0/public/Ticker?pair=ETHXBT).result.XETHXXBT.c.0");
    }

    function __callback(bytes32 myid, string result, bytes proof) {
        if (msg.sender != oraclize_cbAddress()) throw;
        ETHXBT = result;
        // do something with ETHXBT
    }
} 

Answer 2

You can't do this directly; Ethereum contracts can't hit URLs, because Ethereum needs everyone to be able to independently validate the outcome of running any given contract, and you can't guarantee that they'd all get the same result from any given URL.

What you can do is get some third-party, or a combination of third-parties, to hit the URL for you and tell you what they found. They can do this either by signing the data so that your contract can check the signature (this is what we do at Reality Keys) or by sending the data from their contract to your contract (this is what Oraclize do).

Take a look at Etheropt at GitHub for a working example.

The downside of this approach is that your users are required to trust the service that's hitting the URL. If that service is corrupted or hacked, this will result in SFYL. (Oraclize provide a TLS notary proof that the data they provided was the data they supplied you was really supplied by the URL, but that doesn't really help with the actual security risk; Detecting that they gave you the wrong answer is easy, the problem is that your contract is still going to accept their answer even though everybody knows that they're lying...)

There are some potential alternative approaches to this problem that don't rely on reputation; The most famous one is Augur, which gets participants in the market to vote on the outcome and has a system of incentives which they hopes will make people vote truthfully. There are also some interesting proposals in between fully trusted services and pure user voting, like Martin Koeppelmann's "ultimate oracle".

Answer 3

A detailed tutorial for how a contract can obtain data has been written for the Python Ethereum client (pyethapp):

https://github.com/ethereum/pyethapp/wiki/Making-a-User-Service:-Tutorial

One of pyethapp's most powerful distinguishing features is its ability to easily create built-in user services: scripts written in python that run alongside pyethapp, and run code on startup and every time you receive a new block. This allows you to create "server daemons" for applications that require periodic automated support, such as RANDAO, data feeds, "decentralized dropbox" apps, alarm clocks, decentralized cloud computing services, etc.

Pyethapp provides on_start and on_block hooks that allows code to run when it starts up and when it processes a block. Example code from the tutorial:

https://github.com/ethereum/pyethapp/blob/develop/examples/urlfetcher.py

import json, re
import random
import sys
import ethereum.blocks
import ethereum.utils
import ethereum.abi
import rlp
try:
    from urllib.request import build_opener 
except:
    from urllib2 import build_opener

my_privkey = ethereum.utils.sha3('qwufqhwiufyqwiugxqwqcwrqwrcqr')
my_address = ethereum.utils.privtoaddr(my_privkey).encode('hex')
print 'My address', my_address
# Address of the main proxy contract
my_contract_address = ethereum.utils.normalize_address('0xd53096b3cf64d4739bb774e0f055653e7f2cd710')

# Makes a request to a given URL (first arg) and optional params (second arg)
def make_request(*args):
    opener = build_opener()
    opener.addheaders = [('User-agent',
                          'Mozilla/5.0'+str(random.randrange(1000000)))]
    try: 
        return opener.open(*args).read().strip()
    except Exception as e:
        try:
            p = e.read().strip()
        except:
            p = e
        raise Exception(p)


true, false = True, False
# ContractTranslator object for the main proxy contract
ct = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "get(string)", "outputs": [{"type": "int256", "name": "out"}], "inputs": [{"type": "string", "name": "url"}]}, {"inputs": [{"indexed": false, "type": "string", "name": "url"}, {"indexed": false, "type": "address", "name": "callback"}, {"indexed": false, "type": "uint256", "name": "responseId"}, {"indexed": false, "type": "uint256", "name": "fee"}], "type": "event", "name": "GetRequest(string,address,uint256,uint256)"}])
# ContractTranslator object for the contract that is used for testing the main contract
ct2 = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "callback(bytes,uint256)", "outputs": [], "inputs": [{"type": "bytes", "name": "response"}, {"type": "uint256", "name": "responseId"}]}])

app, my_nonce, chainservice = None, None, None

# Called once on startup
def on_start(_app):
    print 'Starting URL translator service'
    global app, my_nonce, chainservice
    app = _app
    chainservice = app.services.chain
    my_nonce = chainservice.chain.head.get_nonce(my_address)


# Called every block
def on_block(blk):
    global my_nonce, chainservice
    for receipt in blk.get_receipts():
        for _log in receipt.logs:
            # Get all logs to the proxy contract address of the right type
            if _log.address == my_contract_address:
                log = ct.listen(_log)
                if log and log["_event_type"] == "GetRequest":
                    print 'fetching: ', log["url"]
                    # Fetch the response
                    try:
                        response = make_request(log["url"])
                    except:
                        response = ''
                    print 'response: ', response
                    # Create the response transaction
                    txdata = ct2.encode('callback', [response, log["responseId"]])
                    tx = ethereum.transactions.Transaction(my_nonce, 60 * 10**9, min(100000 + log["fee"] / (60 * 10**9), 2500000), log["callback"], 0, txdata).sign(my_privkey)
                    print 'txhash: ', tx.hash.encode('hex')
                    print 'tx: ', rlp.encode(tx).encode('hex')
                    # Increment the nonce so the next transaction is also valid
                    my_nonce += 1
                    # Send it
                    success = chainservice.add_transaction(tx, broadcast_only=True)
                    assert success
                    print 'sent tx'

The tutorial explains:

Essentially, an instance of this service run by a trusted party allows Ethereum contracts to access any data feed that is available via a REST API over the internet.

Answer 4

As others have stated, you must use an Oracle. Some projects like ChainLink and Oracleize will do this for you.

To set up an oracle on your own, the process looks like this:

1. Create a smart contract
2. Setup a server listening for some external event, eg a sports game
3. Once a particular condition is met, eg one team wins, your server will send a transaction to your Ethereum contract, updating it with the outcome of the event

Try this example in Javascript (based on Oracles in Ethereum):

Contract

pragma solidity ^0.4.17;

contract GamescoreOracle {
  address public owner;
  string public gameWinner;
  event CallbackGetGameWinner();

  constructor() public {
    owner = msg.sender;
  }

  function updateGameWinner() public {
    emit CallbackGetGameWinner();
  }

  function setWinningTeam(string teamname) public {
    require(msg.sender == owner, "Err: Not Authorized");
    gameWinner = teamname;
  }

  function getGameWinner() public view returns (string) {
    return gameWinner;
  }
}

Oracle server

const fetch = require("fetch");
const OracleContract = require("./build/contracts/GamescoreOracle.json");
const contract = require("truffle-contract");

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.HttpProvider('http://localhost:8545'));

const oracleContract = contract(OracleContract);
oracleContract.setProvider(web3.currentProvider);

web3.eth.getAccounts((err, accounts) => {
  oracleContract.deployed()
    .then((oracleInstance) => {
      // watch event, respond to event with callback
      oracleInstance.CallbackGetGameWinner()
        .watch((err, event) => {
          fetch.fetchUrl('http://www.mocky.io/v2/5baadacb3000002b00a68532', (err, m, response) => {
            const games = JSON.parse(response.toString());
            const winner = games.competitors.filter(team => team.isWinner)[0].team.nickname;

            // Send data back contract on-chain
            console.log(`Running contract.setWinningTeam(${winner})`)
            oracleInstance.setWinningTeam(winner, { from: accounts[0] })
          })
        })
    })
    .catch(err => console.log(err))
})

Answer 5

An oracle can be a manually controlled human account that feeds data in a timely manner or better an automated bunch of bots on traditional servers that scrape a website and feeds in data via the account. You'd need to hard code a contract address into your contract as an oracle of truthful values for your programme. The private key is kept private, unlike a smart contract, where everyone can see it. Some way to de-duplicate and form consensus maybe needed for fully redundant reliable operation.

Answer 6

You have to use an oracle.

An oracle is any device that posts off-chain data to a blockchain.

There are many oracles services which can help you get data from off-chain to on-chain within the Ethereum framework. It's important to note, that each one does it very differently resulting in a lot of considerations for why/how you would implement their technology. Here are some of the most popular:

1. Chainlink
2. Tellor
3. Provable
4. BandChain

And here is an example of getting data through a Chainlink node:

pragma solidity ^0.6.0;

import "github.com/smartcontractkit/chainlink/evm-contracts/src/v0.6/ChainlinkClient.sol";

// MyContract inherits the ChainlinkClient contract to gain the
// functionality of creating Chainlink requests
contract ChainlinkExample is ChainlinkClient {
  // Stores the answer from the Chainlink oracle
  uint256 public currentPrice;
  address public owner;
  
    // The address of an oracle - you can find node addresses on https://market.link/search/nodes
  address ORACLE_ADDRESS = 0xB36d3709e22F7c708348E225b20b13eA546E6D9c;
  // The address of the http get job - you can find job IDs on https://market.link/search/jobs
  string constant JOBID = "628eded7db7f4f799dbf69538dec7ff2";
  // 17 0s = 0.1 LINK 
  // 18 0s = 1 LINK 
  uint256 constant private ORACLE_PAYMENT = 100000000000000000;

  constructor() public {
    setPublicChainlinkToken();
    owner = msg.sender;
  }

  // Creates a Chainlink request with the uint256 multiplier job
  // Ideally, you'd want to pass the oracle payment, address, and jobID as 
  function requestEthereumPrice() 
    public
    onlyOwner
  {
    // newRequest takes a JobID, a callback address, and callback function as input
    Chainlink.Request memory req = buildChainlinkRequest(stringToBytes32(JOBID), address(this), this.fulfill.selector);
    // Adds a URL with the key "get" to the request parameters
    req.add("get", "https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=USD");
    // Uses input param (dot-delimited string) as the "path" in the request parameters
    req.add("path", "USD");
    // Adds an integer with the key "times" to the request parameters
    req.addInt("times", 100);
    // Sends the request with the amount of payment specified to the oracle
    sendChainlinkRequestTo(ORACLE_ADDRESS, req, ORACLE_PAYMENT);
  }

  // fulfill receives a uint256 data type
  function fulfill(bytes32 _requestId, uint256 _price)
    public
    // Use recordChainlinkFulfillment to ensure only the requesting oracle can fulfill
    recordChainlinkFulfillment(_requestId)
  {
    currentPrice = _price;
  }
  
  // withdrawLink allows the owner to withdraw any extra LINK on the contract
  function withdrawLink()
    public
    onlyOwner
  {
    LinkTokenInterface link = LinkTokenInterface(chainlinkTokenAddress());
    require(link.transfer(msg.sender, link.balanceOf(address(this))), "Unable to transfer");
  }
  
  modifier onlyOwner() {
    require(msg.sender == owner);
    _;
  }
  
   // A helper funciton to make the string a bytes32
  function stringToBytes32(string memory source) private pure returns (bytes32 result) {
    bytes memory tempEmptyStringTest = bytes(source);
    if (tempEmptyStringTest.length == 0) {
      return 0x0;
    }
    assembly { // solhint-disable-line no-inline-assembly
      result := mload(add(source, 32))
    }
  }
}

Answer 7

You can get stock, crypto, ETF, etc data from the OrFeed.org smart contract:

examples:

Forex:

uint price = orfeed.getExchangeRate("JPY", "USD", "DEFAULT", 100000);

Stock:

uint price = orfeed.getExchangeRate("AAPL", "USD", "PROVIDER1", 1);

or realtime from a DEX: uint price = orfeed.getExchangeRate("BTC", "DAI", "SELL-UNISWAP-EXCHANGE", 100);