What is the process/workflow behind the process where an Ethereum contract gets some data from a website?

up vote 82 down vote accepted

You can use Oraclize to do exactly that. (It's an example of an Oracle - a mechanism for getting real world information into and out of a contract.)

There are some code examples here as well as the documentation of our Solidity API.

Oraclize is available both on mainnet and testnet so poking around with it should be easy, however if you need any support feel free to ask - we even have a gitter channel here.

As you can see fetching data from a website is as easy as using the oraclize_query function.

Our good behaviour is granted by the TLSNotary proof and can be easily checked with this web based client-side network monitor.

For example to fetch the ETHXBT price from the Kraken ticker:

import "dev.oraclize.it/api.sol";

contract KrakenPriceTicker is usingOraclize {
    string public ETHXBT;

    function PriceTicker() {
        oraclize_setNetwork(networkID_testnet);
        oraclize_setProof(proofType_TLSNotary | proofStorage_IPFS);
        oraclize_query("URL", "json(https://api.kraken.com/0/public/Ticker?pair=ETHXBT).result.XETHXXBT.c.0");
    }

    function __callback(bytes32 myid, string result, bytes proof) {
        if (msg.sender != oraclize_cbAddress()) throw;
        ETHXBT = result;
        // do something with ETHXBT
    }
} 
  • 3
    I had no idea someone was already using TLS Notary for this. Needless to say - mind blown. If you're doing this right, it should allow for trustless information to be passed onto the Ethereum network. (Trustless as long as you trust the server.) – linagee Jan 20 '16 at 17:38
  • 2
    Are there plans to open source Oraclize? (The core software, not the interface.) – linagee Jan 20 '16 at 18:51
  • We are planning to opensource many of the tools we are building at Oraclize, but there is no plan to opensource our internal engine – Thomas Bertani Jan 20 '16 at 20:48
  • 1
    I'm full of questions about Oraclize! I asked one of them here. – Jeff Coleman Jan 21 '16 at 7:54
  • 1
    @Rémi you can find infos around the pricing in our dedicated section at docs.oraclize.it – Thomas Bertani Dec 5 '16 at 20:59

You can't do this directly; Ethereum contracts can't hit URLs, because Ethereum needs everyone to be able to independently validate the outcome of running any given contract, and you can't guarantee that they'd all get the same result from any given URL.

What you can do is get some third-party, or a combination of third-parties, to hit the URL for you and tell you what they found. They can do this either by signing the data so that your contract can check the signature (this is what we do at Reality Keys) or by sending the data from their contract to your contract (this is what Oraclize do).

Take a look at Etheropt at GitHub for a working example.

The downside of this approach is that your users are required to trust the service that's hitting the URL. If that service is corrupted or hacked, this will result in SFYL. (Oraclize provide a TLS notary proof that the data they provided was the data they supplied you was really supplied by the URL, but that doesn't really help with the actual security risk; Detecting that they gave you the wrong answer is easy, the problem is that your contract is still going to accept their answer even though everybody knows that they're lying...)

There are some potential alternative approaches to this problem that don't rely on reputation; The most famous one is Augur, which gets participants in the market to vote on the outcome and has a system of incentives which they hopes will make people vote truthfully. There are also some interesting proposals in between fully trusted services and pure user voting, like Martin Koeppelmann's "ultimate oracle".

A detailed tutorial for how a contract can obtain data has been written for the Python Ethereum client (pyethapp):

https://github.com/ethereum/pyethapp/wiki/Making-a-User-Service:-Tutorial

One of pyethapp's most powerful distinguishing features is its ability to easily create built-in user services: scripts written in python that run alongside pyethapp, and run code on startup and every time you receive a new block. This allows you to create "server daemons" for applications that require periodic automated support, such as RANDAO, data feeds, "decentralized dropbox" apps, alarm clocks, decentralized cloud computing services, etc.

Pyethapp provides on_start and on_block hooks that allows code to run when it starts up and when it processes a block. Example code from the tutorial:

https://github.com/ethereum/pyethapp/blob/develop/examples/urlfetcher.py

import json, re
import random
import sys
import ethereum.blocks
import ethereum.utils
import ethereum.abi
import rlp
try:
    from urllib.request import build_opener 
except:
    from urllib2 import build_opener

my_privkey = ethereum.utils.sha3('qwufqhwiufyqwiugxqwqcwrqwrcqr')
my_address = ethereum.utils.privtoaddr(my_privkey).encode('hex')
print 'My address', my_address
# Address of the main proxy contract
my_contract_address = ethereum.utils.normalize_address('0xd53096b3cf64d4739bb774e0f055653e7f2cd710')

# Makes a request to a given URL (first arg) and optional params (second arg)
def make_request(*args):
    opener = build_opener()
    opener.addheaders = [('User-agent',
                          'Mozilla/5.0'+str(random.randrange(1000000)))]
    try: 
        return opener.open(*args).read().strip()
    except Exception as e:
        try:
            p = e.read().strip()
        except:
            p = e
        raise Exception(p)


true, false = True, False
# ContractTranslator object for the main proxy contract
ct = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "get(string)", "outputs": [{"type": "int256", "name": "out"}], "inputs": [{"type": "string", "name": "url"}]}, {"inputs": [{"indexed": false, "type": "string", "name": "url"}, {"indexed": false, "type": "address", "name": "callback"}, {"indexed": false, "type": "uint256", "name": "responseId"}, {"indexed": false, "type": "uint256", "name": "fee"}], "type": "event", "name": "GetRequest(string,address,uint256,uint256)"}])
# ContractTranslator object for the contract that is used for testing the main contract
ct2 = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "callback(bytes,uint256)", "outputs": [], "inputs": [{"type": "bytes", "name": "response"}, {"type": "uint256", "name": "responseId"}]}])

app, my_nonce, chainservice = None, None, None

# Called once on startup
def on_start(_app):
    print 'Starting URL translator service'
    global app, my_nonce, chainservice
    app = _app
    chainservice = app.services.chain
    my_nonce = chainservice.chain.head.get_nonce(my_address)


# Called every block
def on_block(blk):
    global my_nonce, chainservice
    for receipt in blk.get_receipts():
        for _log in receipt.logs:
            # Get all logs to the proxy contract address of the right type
            if _log.address == my_contract_address:
                log = ct.listen(_log)
                if log and log["_event_type"] == "GetRequest":
                    print 'fetching: ', log["url"]
                    # Fetch the response
                    try:
                        response = make_request(log["url"])
                    except:
                        response = ''
                    print 'response: ', response
                    # Create the response transaction
                    txdata = ct2.encode('callback', [response, log["responseId"]])
                    tx = ethereum.transactions.Transaction(my_nonce, 60 * 10**9, min(100000 + log["fee"] / (60 * 10**9), 2500000), log["callback"], 0, txdata).sign(my_privkey)
                    print 'txhash: ', tx.hash.encode('hex')
                    print 'tx: ', rlp.encode(tx).encode('hex')
                    # Increment the nonce so the next transaction is also valid
                    my_nonce += 1
                    # Send it
                    success = chainservice.add_transaction(tx, broadcast_only=True)
                    assert success
                    print 'sent tx'

The tutorial explains:

Essentially, an instance of this service run by a trusted party allows Ethereum contracts to access any data feed that is available via a REST API over the internet.

An oracle can be a manually controlled human account that feeds data in. Or an automated bot that scrapes a website and feeds in data via the account. You'd need to hard code a contract address into your contract as an oracle of truthful values for your programme.

As others have stated, you must use an Oracle. Some projects like ChainLink and Oracleize will do this for you.

To set up an oracle on your own, the process looks like this:

  1. Create a smart contract
  2. Setup a server listening for some external event, eg a sports game
  3. Once a particular condition is met, eg one team wins, your server will send a transaction to your Ethereum contract, updating it with the outcome of the event

Try this example in Javascript (based on Oracles in Ethereum):

Contract

pragma solidity ^0.4.17;

contract GamescoreOracle {
  address public owner;
  string public gameWinner;
  event CallbackGetGameWinner();

  constructor() public {
    owner = msg.sender;
  }

  function updateGameWinner() public {
    emit CallbackGetGameWinner();
  }

  function setWinningTeam(string teamname) public {
    require(msg.sender == owner, "Err: Not Authorized");
    gameWinner = teamname;
  }

  function getGameWinner() public view returns (string) {
    return gameWinner;
  }
}

Oracle server

const fetch = require("fetch");
const OracleContract = require("./build/contracts/GamescoreOracle.json");
const contract = require("truffle-contract");

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.HttpProvider('http://localhost:8545'));

const oracleContract = contract(OracleContract);
oracleContract.setProvider(web3.currentProvider);

web3.eth.getAccounts((err, accounts) => {
  oracleContract.deployed()
    .then((oracleInstance) => {
      // watch event, respond to event with callback
      oracleInstance.CallbackGetGameWinner()
        .watch((err, event) => {
          fetch.fetchUrl('http://www.mocky.io/v2/5baadacb3000002b00a68532', (err, m, response) => {
            const games = JSON.parse(response.toString());
            const winner = games.competitors.filter(team => team.isWinner)[0].team.nickname;

            // Send data back contract on-chain
            console.log(`Running contract.setWinningTeam(${winner})`)
            oracleInstance.setWinningTeam(winner, { from: accounts[0] })
          })
        })
    })
    .catch(err => console.log(err))
})

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.