MongoDB and Perl

By Fayland Lam on April 26, 2012 11:17 AM | No TrackBacks
it's really a pain to work with MongoDB in Perl. Perl has no 'type' so when you get INT value from DBI, it might be really "1" instead of int 1.

I got some data dumped from MySQL to MongoDB and found all the 'time' field is wrapped as "1335350669" instead of 1335350669.

and when you insert it from code like { time => time() }, you really have 1335350670 instead of "1335350670". it breaks the sort. it breaks the deletion. it breaks everything.

blabla, to fix that, we just need update it in mongodb like below.

PRIMARY> db.jobs.find().forEach(
    function(job) {
        job.time = parseInt(job.time);
        db.jobs.save(job);
    });

but it's still a pain. I didn't check MongoDBx::Class or Mongoose yet, but Moose's type should be able to fix it I think.

Thanks

api and android app

By Fayland Lam on April 14, 2012 8:20 PM | No TrackBacks
I'm a fan of Android. even I like my iPad and MBP very much too.

I'd like to write an App for Android. and here is all the story.

first of all, I need write an API which supports JSONP. since it would be very simple, I don't want to mix it up with Dancer. at last, I wrote something based on webmachine-perl. I like the idea behind the webmachine. the chain design looks pretty nice. even I don't use that too much.

the code is at


and live demo as


after API is done. now comes the android app part. Sorry that I don't know much about Java. and I failed to download appmobi/jqmobi (the download never ends here). so at last I picked up phonegap.

the progress went pretty smooth. mixed with jQuery Mobile, I have it out after few hours.

you can download it for fun from http://static.findmjob.com/FindmJob.apk
it's pretty simple, just with one JSONP request and not much different than the demo.

even there, I'd like to share the code with you:


it's very cool and I'm excited! (badly Google costs 25$ for submitting it and I don't want to do it for now)

Thanks.



sphinx search with varchar primary key

By Fayland Lam on April 8, 2012 9:59 AM | No TrackBacks
usually when you index the mysql data into sphinx, you'll use id int/bigint for the primary key. but it's broken for me on http://findmjob.com/, we use uuid everywhere for the primary key.

here is the solution for it. use @id := @id + 1 for the indexer, and use sql_field_string to get the real id when matched. sample code below:

sql_query_pre = SET NAMES utf8
sql_query_pre = SET @id := 1;
sql_query = \
    SELECT @id := @id + 1 AS tid, id, title, description, location, contact, inserted_at FROM job ORDER BY inserted_at DESC LIMIT 10000

sql_field_string = id

it requires latest sphinx to support sql_field_string. and the latest CPAN module too.

and Perl code for it will be normal like before. and instead you use ->{doc}, you need use the attribtues ->{id} like

my @jobids = map { $_->{id} } @{$ret->{matches}};

so now we have sphinx search supports in my new site, eg: http://findmjob.com/search/Perl.html?q=Perl

Thanks.

DBIx::Class with Moose has

By Fayland Lam on April 4, 2012 9:10 PM | No TrackBacks
well, I don't know how to name it.

DBIx::Class is one of my favorite modules. for its structure, with DBIx::Class you can make all your code very well organized and clean. writing code with DBIx::Class means you can use it in framework like Catalyst or Mojo or Dancer, and you can use it in any perl script (cron usually). for me, DBIx::Class is the right model, TT2 is the right template, and framework is just for URL dispatch.

Moose is another module I like. Role, and clean OO.

by using DBIx::Class::Schema::Loader make_schema_at, with use_moose => 1 on (eg: https://github.com/fayland/findmjob.com/blob/master/script/make_schema_at.pl), we can generate very clean Result module with Moose.

sometimes you want make assessor based on the table column. like for each job in table, we all have a URL which is based on the id and title in the table, it's so related to those two fields, so we'd better to put it in the Result.
with DBIx::Class, it's very simple. (sample code)

package FindmJob::Schema::Result::Job;

....

use FindmJob::Utils 'seo_title';
has 'url' => ( is => 'ro', isa => 'Str', lazy_build => 1 );
sub _build_url {
    my ($self) = @_;

    return "/job/" . $self->id . "/" . seo_title($self->title) . ".html";
}

after that, we can always call $job->url after we ->search for ->find it. very neat. live demo like: 


have fun. Thanks

better pagination url design in Dancer

By Fayland Lam on April 2, 2012 7:23 PM | No TrackBacks
usually People do param for pager like ?page=1 or ?p=1, it maybe not that good for search engine because they may not go scrape inside. so we may come out a solution with /page=1/ or /p=1/ or even /p.1/ etc.

in Dancer, it's very tricky to do add pagination regex in all URLs. and thank God, we have 'forward' and with code like below, it becomes very simple and easy to use.

Perl code:
get qr'.*?/p\.(\d+).*?' => sub {
    my $uri = request->uri;
    $uri =~ s'/p\.(\d+)'';
    var page => $1;
    $uri =~ s/\/$//;
    forward $uri;
};

get '/' => sub {
    my $p = vars->{page} || 1; $p = 1 unless $p =~ /^\d+$/;

we use regex to get the page stuff, then remove it from the request uri then using forward to do a internal request.



have fun.

findmjob.com

By Fayland Lam on April 2, 2012 3:35 PM | No TrackBacks
maybe to earn some money (not for fun this time), I decided to write a new website http://findmjob.com/

I sit front of my computer and coded it for 2 days and here is it. it's out.

it's very simple and without much stuff yet. and the final goal is undecided. but there it is. I'm very pleased to see it in public.

since there is no reason to keep it private, I opened source it in github: https://github.com/fayland/findmjob.com

for programmer, it's very simple to write website. but it's very hard to make it a success. so suggestions are welcome.

Thanks.

Net-GitHub-0.43_01

By Fayland Lam on March 30, 2012 8:43 PM | No TrackBacks
Github is moving on with their API that "We will terminate API v1 and API v2 in 1 month on May 1st, 2012.".

I know lots of people are preferring that they want token instead of writing user/pass in the config or code.
and now we have the choice with create access_token with code instead of web flow.

it's time to kick Net::GitHub default to V3 now. so here comes the beta release. and tests/patches are welcome.


Tips to create access_token with script:

use Net::GitHub;
my $gh = Net::GitHub->new( login => 'fayland', pass => 'secret' );
my $oauth = $gh->oauth;
my $o = $oauth->create_authorization( {
    scopes => ['user', 'public_repo', 'repo', 'gist'], # just ['public_repo']
    note   => 'test purpose',
} );
print $o->{token};

record the token down, and later on we can always use that token without writing down user/pass.

my $github = Net::GitHub->new(
    access_token => $token, # from above
);

Thanks.

Plack::Middleware::FileWrap

By Fayland Lam on December 4, 2011 4:51 PM | No TrackBacks
When you really go coding, you'll meet lots of issues. then you'll write solution for them. that's straight.

Today I have another CPAN module Plack::Middleware::FileWrap out to fit my demand: I'll have lots of plain HTML files, they'll share the same header/footer and I don't want to use stupid iframe.

under Plack, it just means wrap $res->[2] with file content or strings. 

so here comes Plack::Middleware::FileWrap, very simple if you looked at the source code.

then I used it in the KinderGarden project, with snippets:

    mount '/static/docs/' => builder {
        enable 'FileWrap', headers => ["$root/static/docs/header.html"], footers => ["$root/static/docs/footer.html"];
        Plack::App::File->new( root => "$root/static/docs" )->to_app;
    },
    

now all the files under static/docs will wrapped with header.html and footer.html (header.html/footer.html itself too!)

for some advanced example, like if you just want to apply to html files, then you can code something like:

    mount '/static/docs/' => builder {
        enable_if { $_[0]->{PATH_INFO} =~ /\.html/ } 'FileWrap',
            headers => ["$root/static/docs/header.html"], footers => ["$root/static/docs/footer.html"];
        Plack::App::File->new( root => "$root/static/docs" )->to_app;
    },
    
I didn't put the Content-Type check b/c I think it's better to be handled with enable_if.

Enjoy. Thanks

git submodule

By Fayland Lam on December 3, 2011 6:56 PM | No TrackBacks
When you include another open source in your own project, it's usually pretty hard to keep it up to date. it becomes even more harder if you have some modification on it.

recently I have bootstrap in my KinderGarden project, it's very easy to use git submodule to handle it.

kindergarden> git submodule add https://github.com/twitter/bootstrap.git static/bootstrap
kindergarden> git add .gitmodules static/bootstrap
kindergarden> git commit -a -m "remote bootstrap"
kindergarden> git push
kindergarden> git submodule init

in delopy or other machine:

kindergarden$ git submodule init
kindergarden$ git submodule update

pretty easy and simple, and the logic behind is simple too. reference as http://help.github.com/submodules/ or http://progit.org/book/ch6-6.html

BTW, I added Live.com OAuth2 supports to KinderGarden.

Enjoy. Thanks

2011 CN Perl Advent

By Fayland Lam on December 1, 2011 7:16 PM | No TrackBacks
Hi, it's time for advent again!



Enjoy!
Archives