Herschel Archive - Command Line Access

Command Line Access

The entry point is a TAP (Table Access Protocol) server.

You may use HTTP protocol to execute TAP requests at http://archives.esac.esa.int/hsa/whsa-tap-server/tap.

TAP provides two operation modes: Synchronous and Asynchronous:

Synchronous: the response to the request will be generated as soon as the request received by the server.
Asynchronous: the server will start a job that will execute the request. The first response to the request is the required information (a link) to obtain the job status. Once the job is finished, the results can be retrieved.

Our TAP server provides two access mode: public and authenticated:

Public: this is the standard TAP access. A user can execute ADQL queries and upload tables to be used in a query 'on-the-fly' (these tables will be removed once the query is executed). The results are available to any other user and they will remain in the server for a limited space of time.
Authenticated: some functionalities are restricted to authenticated users only The results are saved in a private user space and they will remain in the server for ever (they can be removed by the user).
- ADQL queries and results are saved in a user private area.
- Persintance of uploaded tables: a user can upload a table in a private space.

You can find more about ADQL at:

Here you can find some examples about how to interact with a TAP server (we are using the curl tool):

1. Non authenticated access

1.1. Getting all public tables

curl "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/tables"

1.2. Synchronous query

curl "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+*+FROM+hsa.v_active_observation"

The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.3. Synchronous query on an 'on-the-fly' uploaded table

curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" http://archives.esac.esa.int/hsa/whsa-tap-server/tap/sync

Where 'test_ra_dec.vot' is a file that contains the VOTable to be uploaded (in order to be used by the query)

The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.4. Asynchronous query

curl -i -X POST --data "PHASE=run&LANG=ADQL&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+hsa.v_active_observation" "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async"

Note that there is the possibility to use the optional parameters "JOBNAME" to assign a name to the job and "JOBDESCRIPTION" to add a description:

curl -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+hsa.v_active_observation" "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async"

The response will contain the URL of the job running at server side (see Location header):

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 14:44:39 GMT
Server: Apache-Coyote/1.1
Location: http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404139480755A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked

To obtain the status of the running job:

curl "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404139480755A"

The status response is something like:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <uws:jobId><![CDATA[1404139480755A]]></uws:jobId>
        <uws:runId xsi:nil="true" />
        <uws:ownerId><![CDATA[anonymous]]></uws:ownerId>
        <uws:phase>COMPLETED</uws:phase>
        <uws:quote xsi:nil="true" />
        <uws:startTime>2014-06-30T16:44:40.766+0200</uws:startTime>
        <uws:endTime>2014-06-30T16:44:40.830+0200</uws:endTime>
        <uws:executionDuration>0</uws:executionDuration>
        <uws:destruction>2014-07-07T16:44:40.754+0200</uws:destruction>
        <uws:parameters>
                <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter>
                <uws:parameter id="query"><![CDATA[select top 5 * from hsa.v_active_observation]]></uws:parameter>
                <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter>
                <uws:parameter id="format"><![CDATA[votable]]></uws:parameter>
                <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter>
                <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter>
                <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter>
        </uws:parameters>
        <uws:results>
                <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Farchives.esac.esa.int%2Fhsa%2Fwhsa-tap-server%2Ftap%2Fasync%2F1404139480755A%2Fresults%2Fresult" mime="application/x-votable+xml" size="8863" rows="6" />
        </uws:results>
        <uws:errorSummary xsi:nil="true" />

To obtain the results of the job (once the job is finished):

curl "http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404139480755A/results/result"

The retrieved results is a VO table by default (see '3.3. Asynchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.5. Python script to execute an asynchronous query and wait until the results are available

#ASYNCHRONOUS REQUEST

import httplib
import urllib
#import http.client in Python 3
#import urllib.parse in Python 3
import time
from xml.dom.minidom import parseString

host = "archives.esac.esa.int"
port = 80
pathinfo = "/hsa/whsa-tap-server/tap/async"


#-------------------------------------
#Create job

params = urllib.urlencode({\
	"REQUEST": "doQuery", \
	"LANG":    "ADQL", \
	"FORMAT":  "votable", \
	"PHASE":  "RUN", \
	"JOBNAME":  "Any name (optional)", \
	"JOBDESCRIPTION":  "Any description (optional)", \
	"QUERY":   "SELECT DISTANCE(POINT('ICRS',ra,dec), POINT('ICRS',266.41683,-29.00781)) AS dist, * FROM hsa.v_active_observation  WHERE 1=CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',266.41683,-29.00781, 0.08333333)) ORDER BY dist ASC"
	})

headers = {\
	"Content-type": "application/x-www-form-urlencoded", \
	"Accept":       "text/plain" \
	}

connection = httplib.HTTPConnection(host, port)
connection.request("POST",pathinfo,params,headers)

#Status
response = connection.getresponse()
print "Status: " +str(response.status), "Reason: " + str(response.reason)

#Server job location (URL)
location = response.getheader("location")
print "Location: " + location

#Jobid
jobid = location[location.rfind('/')+1:]
print "Job id: " + jobid

connection.close()

#-------------------------------------
#Check job status, wait until finished

while True:
	connection = httplib.HTTPConnection(host, port)
	connection.request("GET",pathinfo+"/"+jobid)
	response = connection.getresponse()
	data = response.read()
	#XML response: parse it to obtain the current status
	dom = parseString(data)
	phaseElement = dom.getElementsByTagName('uws:phase')[0]
	phaseValueElement = phaseElement.firstChild
	phase = phaseValueElement.toxml()
	print "Status: " + phase
	#Check finished
	if phase == 'COMPLETED': break
	#wait and repeat
	time.sleep(0.2)

#print "Data:"
#print data

connection.close()

#-------------------------------------
#Get results
connection = httplib.HTTPConnection(host, port)
connection.request("GET",pathinfo+"/"+jobid+"/results/result")
response = connection.getresponse()
data = response.read()
outputFileName = "example3_votable_output.vot"
outputFile = open(outputFileName, "w")
outputFile.write(data)
outputFile.close()
connection.close()
print "Data saved in: " + outputFileName

The saved file is a VO table (by default, see '3.3. Asynchronous Queries' section parameters to specify a different output format). The file can be inspected using any analysis tool like TOPCAT, for instance.

2. Authenticated access

2.1. Login

curl -k -c cookies.txt -X POST -d username=USERNAME -d password=PASSWORD -L "https://archives.esac.esa.int/hsa/whsa-tap-server/login"

2.2. Logout

curl -k -b cookies.txt -X POST -d -L "https://archives.esac.esa.int/hsa/whsa-tap-server/logout"

2.3. Getting public and user tables

curl -k -b cookies.txt -X POST -L "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/tables"

2.4. Asynchronous query

curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+hsa.v_active_observation" "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/async"

Note that there is the possibility to use the optional parameters "JOBNAME" to assign a name to the job and "JOBDESCRIPTION" to add a description:

curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&REQUEST=doQuery&QUERY=select+top+5+*+from+hsa.v_active_observation" "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/async"

The response will contain the URL of the job running at server side:

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 15:02:00 GMT
Server: Apache-Coyote/1.1
Location: http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404140520859A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked

To obtain the status of the running job:

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404140520859A"

The status response is something like:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <uws:jobId><![CDATA[1404141177261A]]></uws:jobId>
        <uws:runId xsi:nil="true" />
        <uws:ownerId><![CDATA[USERNAME]]></uws:ownerId>
        <uws:phase>COMPLETED</uws:phase>
        <uws:quote xsi:nil="true" />
        <uws:startTime>2014-06-30T17:12:57.273+0200</uws:startTime>
        <uws:endTime>2014-06-30T17:12:57.864+0200</uws:endTime>
        <uws:executionDuration>0</uws:executionDuration>
        <uws:destruction>2014-07-07T17:23:31.362+0200</uws:destruction>
        <uws:parameters>
                <uws:parameter id="jobdescription"><![CDATA[]]></uws:parameter>
                <uws:parameter id="jobname"><![CDATA[]]></uws:parameter>
                <uws:parameter id="session"><![CDATA[1404141103551]]></uws:parameter>
                <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter>
                <uws:parameter id="query"><![CDATA[SELECT DISTANCE(POINT('ICRS',ra,dec), POINT('ICRS',266.41683,-29.00781)) AS dist, * 
FROM hsa.v_active_observation 
WHERE 1=CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',266.41683,-29.00781, 0.08333333)) ORDER BY dist ASC]]></uws:parameter>
                <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter>
                <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter>
                <uws:parameter id="format"><![CDATA[votable]]></uws:parameter>
                <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter>
                <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter>
        </uws:parameters>
        <uws:results>
                <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Farchives.esac.esa.int%2Fhsa%2Fwhsa-tap-server%2Ftap%2Fasync%2F1404141177261A%2Fresults%2Fresult" mime="application/x-votable+xml" size="2468741" rows="4734" />
        </uws:results>
        <uws:errorSummary xsi:nil="true" />

To obtain the results of the job (once the job is finished):

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/async/1404140520859A/results/result"

2.5. Events capability

Each action (table creation/removal, sharing creation/removal, etc.) generates an event. Events are associated to a session and they are kept while the session (connecton) is open.
Events are grouped by types and each time has a 'latest modification time' (latest time that this type of event was generated).
The following example retrieves the events for a user:

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/event"

The response will contain the events and times associated.

event_type1=timestamp1
event_type2=timestamp2
...

2.6. Notifications capability

Notifications provides information about some activities. Notifications are available for authenticated users only and are kept for a month.
The following example retrieves the notifications for a user:

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/notification?action=GetNotifications"

A response example:

notification_id1[type: notification_type1,notification_subtype1]=explanation1
notification_id2[type: notification_type2,notification_subtype2]=explanation2
...

2.7. Listing jobs

Jobs can be listed using the following request:

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/jobs/list?offset=index&limit=jobs_number&order=order"

Where list specifies the jobs list (e.g. 'sync' or 'async'). It is mandatory.
index specifies the number of jobs to skip before beginning to return the first job. By default, it is '0'.
limit specifies the number of jobs to be returned. No limit by default is set.
order specifies the order of the results.

Example

curl -k -b cookies.txt "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/jobs/async?offset=0&limit=20&order=CREATION_TIME:DESC"

2.8. Deleting jobs

Jobs can be deleted using their identifiers:

curl -k -b cookies.txt -X POST --data "JOB_IDS=job_id1,job_id2..." "https://archives.esac.esa.int/hsa/whsa-tap-server/tap/deletejobs"

3. Interface

See the following specifications:

3.1. TAP resources

http://archives.esac.esa.int/hsa/whsa-tap-server/tap/

Tables	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/tables`
Synchronous access	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/sync`
Asynchronous access	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/async`
Service availability	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/availability`
Events capability	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/event`	TAP+
Notifications capability	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/notification`	TAP+
Jobs listing capability	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/jobs`	TAP+
Jobs removal capability	`http://archives.esac.esa.int/hsa/whsa-tap-server/tap/deletejobs`	TAP+

3.2. Synchronous Queries

Parameter	Value	Comments
REQUEST	doQuery	Requests to execute the provided query
LANG	ADQL	Query language
FORMAT	- votable - votable_plain - csv - json	Results output format
QUERY	ADQL query	query

3.3. Asynchronous Queries

Parameter	Value	Comments
Same parameters as defined in 3.2 Synchronous Queries and
PHASE	run	Query job initial phase

The response header will contain the location of the job.

3.4. Query on an 'on-the-fly' uploaded table

Use a multipart/form-data (see IETF RFC 2388) HTTP POST

Parameter	Value	Comments
Same parameters as defined in 3.2 Synchronous Queries and
UPLOAD	query_table,param:parameter_table_name	query_table: the name of the table used in the query parameter_table_name: HTTP parameter name that points to the table to be uploaded
parameter_table_name	file	file name that contains the table to be uploaded

For instance, in the following request:

curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" http://archives.esac.esa.int/hsa/whsa-tap-server/tap/sync

table_c is the name of the table used in the query: QUERY="select top 5 * from tap_upload.table_c",
table1 is the name of the HTTP parameter that provides the file: table1=@test_ra_dec.vot and test_ra_dec.vot is the file name that contains the table to be uploaded.

3.5. TAP+ login

Parameter	Value	Comments
username	user_name	User name
password	user_password	User password

The response header will contain the session identifier.

3.6. TAP+ logout

Parameter	Value	Comments
session identifier	session identifier	Session identifier provided by a login request Must be added to the HTTP header

3.7. TAP+ tables capabilities

In addition to the standar 'tables' TAP capability, the following parameters can be used too:

Parameter	Value	Comments
tables	comma separated full qualified table names	A lis of the specified tables will be returned
schemas	comma separated schema names	A list of the specified schemas will be returned
only_tables	TRUE / FALSE (default: FALSE)	TRUE: no columns information will be returned
only_schemas	TRUE / FALSE (default: FALSE)	TRUE: no tables nor columns information will be returned

These parameters are handled based on the following priorities

Priority	Parameter	Comments
1.	tables != null	No more checks are performed (share_info and share_accessible are handled if present)
2.	tables == null (default)	More checks are performed
2.1.	only_schemas = TRUE	No more checks are performed
2.2.	only_schemas = FALSE (default)	The following checks are performed
2.2.1.	schema_names != null	The following parameters are applied to the specified schemas only
2.2.2	only_tables = TRUE	No columns data are generated
2.2.3	only_tables = FALSE (default)	Columns data are generated

3.8. TAP+ events capabilities

Tables creation/removal/sharing actions generate events. Events are grouped by types and each type has a 'last modification time' associated. Those events and times can be retrieved using:

Parameter	Value	Comments
id	event type identifier	Optional parameter. If not provided, all events are returned. If provided, only the requested type is returned.

The current event types are:

Value	Comments
100	Job created
101	Job updated
102	Job removed
210	Shared item created
211	Shared item updated
212	Shared item removed
220	Shared group created
221	Shared group updated
222	Shared group removed
230	Shared user created
231	Shared user updated
232	Shared user removed
300	Log in
301	Log out
401	Database quota updated
402	File quota updated
500	Notification created
501	Notification removed
4000	Table created
4001	Table updated
4002	Table removed

3.9. TAP+ notifications

Notifications are kept for one month. Each notification provides a message that explains the notification. Notifications are available for authenticated users only.

Parameter	Value	Comments
session identifier	session identifier	Session identifier provided by a login request Must be added to the HTTP header
action	GetNotifications	Notificaions associated to the user

3.10. TAP+ jobs capabilities

Parameter	Value	Comments
session identifier	session identifier	Session identifier provided by a login request Must be added to the HTTP header
list	job list identifier (e.g. 'sync'/'async')	Mandatory
offset	Number of jobs to skip	Optional. '0' by default.
limit	Number of jobs to return	Optinal. No limit by default.
order	List of column order records.	Optiona. Each record is composed of: column_name:ASC\|DESC

3.11. TAP+ deletejobs capabilities

Parameter	Value	Comments
session identifier	session identifier	Session identifier provided by a login request Must be added to the HTTP header
JOB_IDS	comma separated job identifiers	Mandatory