What is curl?

What is curl?

cURL, often just “curl,” is a free command line tool. It uses URL syntax to transfer data to and from servers. curl is a widely used because of its ability to be flexible and complete complex tasks. For example, you can use curl for things like user authentication, HTTP post, SSL connections, proxy support, FTP uploads, and more! You can also do simple things with curl, such as download web pages and web images. Read on to find out if you should use curl, and if so, common use cases that will get you started. 

Should You Use curl?

Whether or not you should use curl depends on your goals. For simpler goals, you may want to check out wget. curl is great for complex operations since it is scriptable and versatile. However, wget is easier to understand and more user-friendly, so we recommend using it for simpler tasks.

curl Protocols

curl has many different supported protocols. However, curl will use HTTP protocol by default if no protocol is provided. For example, if you run the following example, it would download the homepage of example.com.

curl example.com

You can call a specific protocol by prefacing the URL with the protocol name.

curl http://example.com

The example above uses the HTTP protocol. If you want to use a different protocol, switch HTTP out for another. For example, if you wanted to use the FTP protocol, it would look like this:

curl ftp://example.com

curl will also try different protocols if the default protocol doesn’t work. If you give it hints, curl can guess what protocol you want to use.

For example, if you wrote the following command, curl would be able to intelligently guess that you wanted to use the FTP:// protocol.

curl ftp.example.com

Here’s a list of curl supported protocols:

Basics: How to Use curl

We touched on how to use curl protocols briefly, which may have given you some idea on how to use curl. At its most basic, curl tends to follow this format:

curl [option] [url] 

You can find a list of possible options on the curl documentation site. Options will direct curl to perform certain actions on the URL listed. The URL gives curl the path to the server it should perform the action on. You can list one URL, several URLs, or parts of a URL, depending on the nature of your option.

Listing more than one URL:

curl -O http://url1.com/file1.html -O http://url2.com/file2.html

Listing different parts of a URL:

http://example.{page1,page2,page3}.html

Saving URL to File

You can also use curl to save the content of the URL to a file. There are two different methods for doing this: the -o and the -O method. When you use the -o option, you can add a filename that the URL will be saved as. A command using the -o option would look something like this:

curl -o filename.html http://example.com/file.html

Notice that the filename the URL will be saved in is placed between the -o option and the URL.

The -O method allows you to save the file under the same name as the URL. When using the -O option, no filename is required between the option and the URL. Instead, the command will look something like this:

curl -O http://example.com/file.html

Continuing a Download

If your download is stopped, you can restart it again with a simple curl command. It’s very simple, all you need to do is rewrite the command with the addition of the -C option.

If you were saving a URL, but the process was halted, you can restart the process by typing in the following:

curl -C -O http://example.com/file.html

This would pick the process back up where it had halted before.

Specify Time Frame for Download

Download files before or after a certain time by using curl.  To do this, use the -z option, then list the date.

curl -z 25-Jan-18 http://example.com

The -z option will search for files after the designated time frame by default. To search for files before the time listed, you can add a dash in front of the date. It will look like this:

curl -z -25-Jan-18 http://example.com

Showing curl Output

curl will often not show any output after you have executed a command, which can be frustrating if you are trying to learn the ropes. The good news? curl has an option that allows you to view curl as it works.

You just need to add a -v to the command to view curl’s internal runnings as it executes. This can be especially helpful when you receive a response from curl that you didn’t anticipate. By viewing curl with -v, you can see what curl is actually doing behind the scenes. Simply run the command to turn it on.

Here’s an example of what a command with the -v option would look like:

curl -v http://example.com

If you get tired of seeing the internal workings of curl, you can also turn this feature off by using the –no-verbose option. Just switch the -v option out for –no-verbose, and curl will stop showing the internal process.

curl --no-verbose http://example.com

curl in Review

curl is a powerful, flexible tool. The commands touched on here were only the tip of the iceberg – curl has the ability to work with a multitude of protocols and, while we only touched on HTTP-specific options. Stay tuned for more blog posts about curl in the future. You can be notified as soon as a new blog post comes out.

If you like the idea of curl, but think it might be too complex for you, check out our article on using wget.