setting HTTP Headers in apache httpd.conf

In a lot of cases, we have to set particular http headers for objects. For example to set the caching age, caching control etc. How the http headers can be set? This can be done very easily using the Apache web server configuration file.


Suppose you want to set http headers for the mp4 or mpd(MPEG Dash) video files served by the webserver.

Open the httpd.conf in your favourite editor and add the following.

<FilesMatch “.(mpd|mp4)”>
        Header set Access-Control-Allow-Origin “*”
        Header set Access-Control-Allow-Headers “Origin,Content-Type,Access-Control-Allow-Origin,Range,Accept”

        Header set Access-Control-Expose-Headers “Access-Control-Allow-Origin, Content-Length, Content-Type, Date, Range, Server, Transfer-Encoding”
        Header set Access-Control-Allow-Methods “GET,POST,PUT,DELETE,OPTIONS”

        Header set Access-Control-Max-Age “1800”
</FilesMatch>

Restart or reload Apache.

Checking the response:
Response:
[root@localhost ~]# curl -I http://xxx.amazonaws.com/DARK_KNIGHT.mpd
HTTP/1.1 200 OK
Date: Wed, 20 May 2015 08:36:17 GMT
Server: Apache
Last-Modified: Mon, 04 May 2015 13:16:44 GMT
ETag: “682461-a5a-5f91c700”
Accept-Ranges: bytes
Content-Length: 2650
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Origin,Content-Type,Access-Control-Allow-Origin,Range,Accept
Access-Control-Expose-Headers: Access-Control-Allow-Origin, Content-Length, Content-Type, Date, Range, Server, Transfer-Encoding
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Max-Age: 1800
Connection: close
Content-Type: text/xml

Thats it. Have fun. Share the knowledge. 

uses of curl linux command with examples

According to various definitions in the internet, curl is a tool to transfer data from or to a server. Curl supports the following protocols. DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. I was working with content delivery networks(cdn) and came across this command to get the headers related to the caching of the objects. Later when I checked the use cases, I found it very interesting. So I’m sharing the points here.
Starting with the cdn related commands.

if we use -v option with curl with the url of the object (in this case cdn we are using in example is amazon cloufront), it will give all the request headers and response headers.

randeep@Randeep:~$ curl -v http://abcdefghi.cloudfront.net/images/CLogo_bollywood_192.png
* Hostname was NOT found in DNS cache
*   Trying 54.230.190.xxx…
* Connected to abcdefghi.cloudfront.net (54.230.190.xxx) port 80 (#0)
> GET /images/CLogo_bollywood_192.png HTTP/1.1
> User-Agent: curl/7.35.0
> Host: abcdefghi.cloudfront.net
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/png
< Content-Length: 8046
< Date: Mon, 16 Feb 2015 02:04:03 GMT
< Last-Modified: Wed, 25 Jun 2014 12:10:59 GMT
< ETag: “530dd4308aaedc70a3aae9b0889bd”
< Accept-Ranges: bytes
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
< Age: 203184
< X-Cache: Hit from cloudfront
< Via: 1.1 29b3cfc63bec85046291502a4.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: EGjPC9oHhVYc6gdczCygXeAGJtPGIuYrklTMaNlko98RQ==
< Connection: keep-alive
<
Followed by PNG data.

In case if you want to see only the headers in the response, you can use -I option with curl

randeep@Randeep:~$ curl -I http://abcdefghi.cloudfront.net/images/CLogo_bollywood_192.png
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 8046
Date: Mon, 16 Feb 2015 02:04:03 GMT
Last-Modified: Wed, 25 Jun 2014 12:10:59 GMT
ETag: “530dd4308aaedc7063aae9b0889bd”
Accept-Ranges: bytes
Server: AmazonS3
Age: 203184
X-Cache: Hit from cloudfront
Via: 1.1 29b3cfc63bec85091502a4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: EGjPC9oHhV6gdczCygX7qEPGIuYrklTMaNlko98RQ==
Connection: keep-alive

Checking whether the content can be cached or not:
In this example content can be cached.
randeep@Randeep:~/aws$ curl -H “Cache-control: only-if-cached” -D – -o /dev/null “http://54.161.158.165:8080/

videos/abc.mp4″
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 –:–:– –:–:– –:–:–     0HTTP/1.1 200 OK
Date: Sun, 12 Apr 2015 12:18:41 GMT
Server: Apache/2.2.15 (CentOS)
Last-Modified: Wed, 30 Apr 2014 04:02:06 GMT
ETag: “3b32-1df0d06-4f83a9c9a6b80”
Accept-Ranges: bytes
Content-Length: 31395078
Cache-Control: max-age=604800, public
Connection: close
Content-Type: video/mp4

 14 29.9M   14 4568k    0     0   371k      0  0:01:22  0:00:12  0:01:10  255k
 15 29.9M   15 4787k    0     0   334k      0  0:01:31  0:00:14  0:01:17  128k^C

In this example, content is not cached and cant be.
randeep@Randeep:~/aws$ curl -H “Cache-control: only-if-cached” -D – -o /dev/null “http://54.161.158.165/videos/abc.mp4”  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 –:–:– –:–:– –:–:–     0HTTP/1.1 504 Not Cached
Date: Sun, 12 Apr 2015 12:21:37 GMT
Connection: keep-alive
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 340
100   340  100   340    0     0  46008      0 –:–:– –:–:– –:–:– 48571
randeep@Randeep:~/aws$

sending multiple headers in curl: 
curl -H “Cache-control: only-if-cached” -H “Range: bytes=0-1024” -D – -o /dev/null “http://tata.cdn.bitgravity.com/abc.mp4”

Other interesting uses of curl:

Fetching/Downloading files:
To download a file

Syntax:
$ curl -O URL
Example:
$ curl -O http://www.apache.org/dyn/closer.cgi/trafficserver/trafficserver-5.2.0.tar.bz2
The above example will download the archive file named trafficserver-5.2.0.tar.bz2 in the current directory.

To download multiple files

Syntax:
$ curl -O URL1 -O URL2
Example:
$ curl -O http://www.apache.org/dyn/closer.cgi/trafficserver/trafficserver-5.2.0.tar.bz2 -O http://apache.bytenet.in/tomcat/tomcat-8/v8.0.18/bin/apache-tomcat-8.0.18.tar.gz
The above command will download both trafficserver-5.2.0.tar.bz2, apache-tomcat-8.0.18.tar.gz

To download a file with restricted network speed

Syntax:
$ curl –limit-rate 1000B -O http://download.thinkbroadband.com/1GB.zip
Example:
randeep@Randeep:~$ curl –limit-rate 1000B -O http://download.thinkbroadband.com/1GB.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1024M    0 45455    0     0   1005      0  12d 08h  0:00:45  12d 08h  1229
The above command will download the file 1GB.zip but within the speed limit of 1000Bytes per second.

FTP:
FTP is file transfer protocol.
Downloading a file from ftp:
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php

Uploading a file to ftp
$ curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com

sending mail using curl:
Using curl, we can also send mails. All we need is an smtp server and user credentials.

curl –url “smtps://smtp.example.com:465” –ssl-reqd   –mail-from “me@example.com” –mail-rcpt “you@example.com”   –upload-file content.txt –user “me@example.com:mypassword” –insecure

 That’s it. Read more about curl and comment if you find something very useful.

Various usage of find command can be found here.

Writing logs from the port to a custom file in syslog-ng

We are running a log server (syslog-ng). We are receiving the logs from our hardware devices on port 514. The problem is the logs were writing into “/var/log/messages”. How can we write this log to a separate dedicated file. Also forwarding this log to an another log server? Its better to have a centralized log server which collects logs from all of the other ssytems. So if even one system fails we get the information from the central log server.

We will see how:
 in a syslog-ng-2.1.4-9.el5 server, listening on port 514

Open the syslog-ng configuration file for editing:
[root@logs ~]# vim /etc/syslog-ng/syslog-ng.conf

#Declaring the input
source s_sys {
        udp(ip(0.0.0.0) port(514));
};

#Declaring the destination file
destination d_custom { file(“/var/log/devicemessages”); };

#Writing the rule to forward the input to file  
log { source(s_sys); filter(f_default); destination(d_custom); };

#For forwarding this log to a different log server
destination graylog2 { udp(“xxx.xxx.xxx.xxx” port(514)); };
log {
        source(s_sys);
        destination(graylog2);
};
Best Reads:
1.  Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management