Use EvalAI through command line

Make submissions to a challenge with a single command

Check the real-time status of your submissions through your terminal

See Examples Fork me on GitHub

What is EvalAI-CLI?

EvalAI-CLI is designed to extend the functionality of the EvalAI web application to your command line to make the platform more accessible and terminal-friendly to its users.

Get Started

Install EvalAI from PyPI


Run this command
pip install evalai

Configure your Auth Token


Run this command
evalai set_token <auth_token>

View your Auth Token


Run this command
evalai get_token

Examples


Challenge

Display all challenges

Run this command
evalai challenges

Display all ongoing challenges

Run this command
evalai challenges ongoing

Display all upcoming challenges

Run this command
evalai challenges future

Display all past challenges

Run this command
evalai challenges past

Display details of a challenge with ID 1

Run this command
evalai challenge 1

Display all challenges that you've participated in

Run this command
evalai challenges --participant

Display all challenges that you've hosted

Run this command
evalai challenges --host

Challenge Phase

Display all challenge phases of the challenge with Challenge ID 1

Run this command
evalai challenge 1 phases

Display the details of the phase with ID 4 of the challenge with ID 1

Run this command
evalai challenge 1 phase 4

Teams

Participate in a challenge with ID 1 and team ID 2

Run this command
evalai challenge 1 participate 2

View your participant teams

Run this command
evalai teams --participant

View your host teams

Run this command
evalai teams --host

Create a participant team

Run this command
evalai teams create participant

Create a host team

Run this command
evalai teams create host

Leaderboards and Submissions

Make a submission to the Phase 4 of a challenge with ID 1

If Challenge 1 is Docker based

Run this command
evalai push <image:tag> -p 4

If Challenge 1 is Docker based and you want to make private submission

Run this command
evalai push <image:tag> -p 4 --private

If Challenge 1 is Docker based and you want to make public submission

Run this command
evalai push <image:tag> -p 4 --public

If Challenge 1 is not Docker based

Run this command
evalai challenge 1 phase 4 submit --file submission.json

If Challenge 1 is not Docker based and submission file is > 400 MB

Run this command
evalai challenge 1 phase 4 submit --file submission.json --large

If Challenge 1 is not Docker based and you want to make private submission

Run this command
evalai challenge 1 phase 4 submit --file submission.json --private

If Challenge 1 is not Docker based and you want to make public submission

Run this command
evalai challenge 1 phase 4 submit --file submission.json --public

View status of a submission with ID 78

Run this command
evalai submission 78

Get all the phase splits of the challenge 1 with phase 4

Run this command
evalai challenge 1 phase 4 splits

View the leaderboard of Challenge 1 with Challenge Phase Split 4

Run this command
evalai challenge 1 leaderboard 4

View all the submissions to Phase 4 of the challenge 1

Run this command
evalai challenge 1 phase 4 submissions

View all the submissions to Phase 4 of the challenge 1 between 06/15/18 and 06/25/18

Run this command
evalai challenge 1 phase 4 submissions -s 06/15/18 -e 06/25/18

Challenge annotations

Upload challenge phase annotations if user is challenge host

Run this command
evalai challenge 1 phase 4 submit --file annotation.json --annotation

Upload challenge phase annotations > 400MB if user is challenge host

Run this command
evalai challenge 1 phase 4 submit --file annotation.json --large --annotation

Configure your custom backend

Configure the Host URL (OPTIONAL)

Only run the following command if you want to use CLI with your forked version of EvalAI.

Run this command
evalai host -sh https://eval.ai