How to download “past broadcasts” from twitch.tv

08/08/2013 2 comments

Unfortunately justin api no longer support twitch.tv channels and new twitch api doesn’t show direct links for flv videos. But fortunately we can still use listing by archive id using this url:

http://api.justin.tv/api/broadcast/by_archive/yyyyyyyyy.json

So, lets adopt our previous source to get fully working code in php witch download all videos from our favorite twitch channel:

<?php
$url = "https://api.twitch.tv/kraken/channels/xxxxxxx/videos?limit=100&broadcasts=true";

$c = curl_init($url);
$options = array(
	CURLOPT_RETURNTRANSFER => true,
	CURLOPT_HTTPHEADER => array('Content-type: application/json'),
	CURLOPT_SSL_VERIFYPEER => false,
    CURLOPT_SSL_VERIFYHOST => 2
);
curl_setopt_array( $c, $options );
$json = curl_exec($c);
$json_a = json_decode($json,true);

foreach($json_a['videos'] as $v) {
	$id = $v['_id'];
	$id = preg_replace('/[^0-9.]+/', '', $id);
	if(!history_check($id)){
		$vid = "http://api.justin.tv/api/broadcast/by_archive/$id.json";
		$d = curl_init($vid);
		$options2 = array(
			CURLOPT_RETURNTRANSFER => true,
			CURLOPT_HTTPHEADER => array('Content-type: application/json')
		);
		curl_setopt_array( $d, $options2 );
		$json2 = curl_exec($d);
		$json2_a = json_decode($json2,true);
		exec('wget --progress=dot:mega -c '.$json2_a[0]['video_file_url'].' -O "downloads/'.$json2_a[0]['start_timestamp'].' - '.format_title($json2_a[0]['title']).'.flv"', $array,$exit_code);
		echo 'Exit code: '.$exit_code."\n";
		echo $array;
		echo "\n";
		history_add($id);
		echo implode('"\n"', $array);
	}
}

function format_title($title){ //format title and limit to 50 chars
	$out = preg_replace('/[^a-zA-Z \.]/','', $title);
	$out = substr($out, 0, 50); 
	return $out;
}
function history_check($date){
	$data = file_get_contents('history');

	if(strpos($data, $date) !== FALSE)
	{
		$out = TRUE;
	}
	else
	{
		$out = FALSE;
	}
return $out;
}

function history_add($date){
	file_put_contents('history', $date."\n", FILE_APPEND);
}
?>
Categories: Uncategorized

Bebiko Junior competition (bebiklub.pl)

16/07/2013 Leave a comment

Friend of mine participate in competition on bebiklub.pl – just a simple vote system. Unfortunately I can’t see on which place she is, there is no such a functionality, so I need to implement it by myself. As usual I choice php for that:

<?php
for($z=0;$z<=2960;$z=$z+10){
$url='https://www.bebiklub.pl/home/nowe-bebiko-junior/odpowiedzi-konkursowe/?start='.$z;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
$in = curl_exec($ch);
curl_close($ch);

preg_match_all('/data-votes="(.*)" data-name="(.*)" data-id/i', $in, $out);
$k=count($out[1]);
for($i=0;$i<=$k-1;$i++){
        echo $out[2][$i].';';
        echo $out[1][$i]."\n";
}
}
?>

It will output a list of nicknames and votes in format:
nick;number_of_votes
Afterwards that output can be imported to excel and sorted.

Btw. Don’t forget to vote!

Categories: Bash

Backup remote server using tar and ssh

13/07/2013 Leave a comment
ssh user@server "sudo tar cvzp - / --same-owner --exclude=/proc/* --exclude=/media/* --exclude=/dev/* --exclude=/mnt/* --exclude=/sys/* --exclude=/tmp/* --ignore-failed-read" > backup.tgz

user should have sudo rights without password on remote machine.

Categories: Linux, Useful commands

Godziny otwarcia basenu w Malborku

25/04/2013 Leave a comment

Poniedziałek: 7:00, 12:40
Wtorek: 7:00, 13:35
Środa: 7:00
Czwartek: 7:00, 14:30
Piątek: 7:00
Sobota: 11:00, 12:00, 13:00, 14:00, 15:00, 16:00, 17:00
Niedziela: 11:00, 12:00, 13:00, 14:00, 15:00, 17:00

Stan na 2013-04-25

Categories: Uncategorized

How to get fresh file/mysql backup from remote on minimal effort.

19/02/2013 Leave a comment

Lets assume that we have a remote web server (with ssh access) and we want fresh backup of files and mysql databases. Of course we can login, type some commands and then sftp those files but lets do this on minimum effort:

Backup files (tar, gzip and ‘download’ them):

ssh user2@remote "tar cf - /var/www/somepage | gzip -c" > files.tar.gz

Backup MySQL database:

ssh user2@remote "mysqldump -pXXXXXX database | gzip -c" > mysql.gz

Lets merge those command and write a little bash script that will automate those activities:

#!/bin/bash
now=$(date +"%Y%m%d%H%M")
bckdir="/home/user/backups"
ssh user2@remote "mysqldump -pXXXXXX databases | gzip -c" > $bckdir/$now.mysql.gz
ssh user2@remote "tar cf - /var/www/somepage | gzip -c" > $bckdir/$now.files.tar.gz

where:
XXXXXXX is a mysql password,
database is our db to backup.

Categories: Bash, Linux, Useful commands

How to download archived videos from twitch.tv/justin.tv

19/02/2013 1 comment

Working source here.

This php script will download all archived videos from twitch.tv (and probably from justin.tv – because there is same api as on twitch) using wget. All videos would be saved to download folder (create it first) with date as a name and ‘show title’. File name is limited to 50 chars.

Downloading is done by parsing json files from justin, more info: Channel/archives api wiki.

<?php
$url = "http://api.justin.tv/api/channel/archives/xxxx.json?limit=100"; //where xxxx is a channel id or login
$c = curl_init($url);
$options = array(
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => array('Content-type: application/json')
);
curl_setopt_array( $c, $options );
$json = curl_exec($c);
$json_a = json_decode($json,true);
$json_a = array_reverse($json_a);
 
foreach($json_a as $k => $v) {
    if(!history_check(format_date($v['created_on']))){
        exec('wget -c '.$v['video_file_url'].' -O "downloads/'.format_date($v['created_on']).' - '.format_title($v['title']).'.flv"', $array);
        history_add(format_date($v['created_on']));
        echo implode('"\n"', $array);
    }
}
 
function format_date($date){
    $out = preg_replace('/[^0-9\.]/', '', $date);
    return $out;
}
function format_title($title){ //format title and limit to 50 chars
    $out = preg_replace('/[^a-zA-Z \.]/','', $title);
    $out = substr($out, 0, 50);
    return $out;
}
function history_check($date){
    $data = file_get_contents('history');
 
    if(strpos($data, $date) !== FALSE)
    {
        $out = TRUE;
    }
    else
    {
        $out = FALSE;
    }
return $out;
}
 
function history_add($date){
    file_put_contents('history', $date."\n", FILE_APPEND);
}
?>

Usage:
Create download dir, save this source as get.php file and simply run it from command line (php-cli need to be installed):

php get.php

Take in mind that information about downloaded videos is stored in history file. This will avoid downloading videos which was downloaded before.

Run command on each file in directory

19/02/2013 Leave a comment

Most useful when you have many files in directory and bash doesn’t want to cooperate:

-bash: -: Argument list too long

First one:

for file in *; do ls -al "$file"; done

where * could be a file name or extension ex.: *.jpg

And much more easy to remember:

find . -exec ls -al {} \;
Categories: Bash, Linux, Useful commands
Follow

Get every new post delivered to your Inbox.