Hey guys, am josh in Uganda. i created a mnesia fragmented table (64 fragments), and managed to populate it upto 9948723 records. Each fragment was a disc_copies type, with two replicas. Now, using qlc (query list comprehension), was too slow in searching for a record, and was returning inaccurate results.
I found out that this overhead is that qlc uses the select function of mnesia which traverses the entire table in order to match records. i tried something else below.
-define(ACCESS_MOD,mnesia_frag). -define(DEFAULT_CONTEXT,transaction). -define(NULL,'_'). -record(address,{tel,zip_code,email}). -record(person,{name,sex,age,address = #address{}}).
match()-> Z = fun(Spec) -> mnesia:match_object(Spec) end,Z.
match_object(Pattern)-> Match = match(), mnesia:activity(?DEFAULT_CONTEXT,Match,[Pattern],?ACCESS_MOD).
Trying this functionality gave me good results. But i found that i have to dynamically build patterns for every search that may be made in my stored procedures.
i decided to go through the havoc of doing this, so i wrote functions which will dynamically build wild patterns for my records depending on which parameter is to be searched.
%% This below gives me the default pattern for all searches ::= {person,'','','_'}
pattern(Record_name)-> N = length(my_record_info(Record_name)) + 1, erlang:setelement(1,erlang:make_tuple(N,?NULL),Record_name).
%% this finds the position of the provided value and places it in that %% position while keeping '_' in the other positions. %% The caller function can use this function recursively until %% it has built the full search pattern of interest
pattern({Field,Value},Pattern_sofar)-> N = position(Field,my_record_info(element(1,Pattern_sofar))), case N of -1 -> Pattern_sofar; Int when Int >= 1 -> erlang:setelement(N + 1,Pattern_sofar,Value); _ -> Pattern_sofar end.
my_record_info(Record_name)-> case Record_name of staff_dynamic -> record_info(fields,staff_dynamic); person -> record_info(fields,person); _ -> [] end.
%% These below,help locate the position of an element in a list %% returned by "-record_info(fields,person)"
position(_,[]) -> -1; position(Value,List)-> find(lists:member(Value,List),Value,List,1).
find(false,,,_) -> -1; find(true,V,[V|],N)-> N; find(true,V,[|X],N)-> find(V,X,N + 1).
find(V,[V|],N)-> N; find(V,[|X],N) -> find(V,X,N + 1).
This was working very well though it was computationally intensive. It could still work even after changing the record definition since at compile time, it gets the new record info
The problem is that when i initiate even 25 processes on a 3.0 GHz pentium 4 processor running WinXP, It hangs and takes a long time to return results.
If am to use qlc in these fragments, to get accurate results, i have to specify which fragment to search in like this.
find_person_by_tel(Tel)-> select(qlc:q([ X || X <- mnesia:table(Frag), (X#person.address)#address.tel == Tel])).
select(Q)-> case ?transact(fun() -> qlc:e(Q) end) of {atomic,Val} -> Val; {aborted,_} = Error -> report_mnesia_event(Error) end.
Qlc was returning [], when i search for something yet when i use match_object/1 i get accurate results. I found that using match_expressions can help.
mnesia:table(Tab,Props). where Props is a data structure that defines the match expression, the chunk size of return values e.t.c
I got a problem when i tried building match expressions dynamically.
Function mnesia:read/1 or mnesia:read/2 requires that you have the primary key
Now am asking myself, how can i efficiently use QLC to search for records in a large fragmented table? Please help.
I know that using tuple representation of records makes code hard to upgrade. This is why i hate using mnesia:select/1, mnesia:match_object/1 and i want to stick to QLC. QLC is giving me wrong results in my queries from a mnesia table of 64 fragments even on the same node.
Has anyone ever used QLC to query a fragmented table?, please help