I have a very large matrix (100M rows by 100M columns) that has a lots of duplicate values right next to each other. For example: 8 8 8 8 8 8 8 8 8 8 8 8 8 8 4 8 8 1 1 1 1 1 8 8 8 8 8 4 8 8 1 1 1 1 1 8 8 8 8 8 4 8 8 1 1 1 1 1 8 8 8 8 8 4 8 8 1 1 1 1 1 8 8 8 8 8 4 8 8 1 1 1 1 1 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 3 3 3 3 3 3 3 3 3 3 3...
Hello all, I am looking at solving a problem that is a PDE, and the 3D discretized domain can have a different boundary condition on each of the 6 boundaries (or all the same). What is the best way to put this sparse matrix into a compressed format? Is CSR going to be my only option here? I thought about using ellpack, but I am not sur...
Hello all, I have a sparse matrix that represents a 3D rectangular space. Along some of the boundaries, I know what the value is going to be (it's a constant). The other boundaries may be reflective, differential, etc. Should I just set the problem up as if all the boundaries were say, differential, and then go back and set the nodes ...
I'm not sure if this is a good question or not — please close it if not. I set out to write (using boost::coordinate_vector as a starting point) a sparse_vector template class that efficiently implements a vector-like interface, but is sparse. It implements all the usual vector operations and a fast sparse iterator that iterates over t...
I am working on some software that does a sparse matrix-vector multiply. The matrix is stored in a coordinate format (a row and column index for each non-zero). They are performing a sort operation that sorts the column index in order for that row (that is because of some boundary conditions for this FEM problem). Is there some necessit...
Hello again, I am trying to do some (k-means) clustering on a very large matrix. The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of "1" values per row). I want to get around 2000 clusters. I got two questions: - Can someone recommend an open source platform or tool for doing that (maybe using k-means...
Hello, I am using sparse matrices as a mean of compressing data, with loss of course, what I do is I create a sparse dictionary from all the values greater than a specified treshold. I'd want my compressed data size to be a variable which my user can choose. My problem is, I have a sparse matrix with alot of near-zero values, and what I...
Does anyone know how to perform svd operation on a sparse matrix in python? it seems that their is no such functionality provided in scipy.sparse.linalg. ...
Suppose I have a 2d sparse array. In my real usecase both the number of rows and columns are much bigger (say 20000 and 50000) hence it cannot fit in memory when a dense representation is used: >>> import numpy as np >>> import scipy.sparse as ssp >>> a = ssp.lil_matrix((5, 3)) >>> a[1, 2] = -1 >>> a[4, 1] = 2 >>> a.todense() matrix([[...
In Matlab, at what point is having a sparse array better than a normal array if I still have a lot of calculations to do on it, and about 25% of the array are non-zeros? ...
I am implementing a sparse matrix class in compressed row format. This means i have a fixed number of rows and each row consists of a number of elements (this number can be different for different rows but will not change after the matrix has been initialised. Is it suitable to implement this via vectors of vectors or will this somehow ...
Are there any algorithms that allow efficient creation (element filling) of sparse (e.g. CSR or coordinate) matrix in parallel? ...
What do you think? What would be faster and how much faster: Doing sparse matrix (CSR) multiplication (with a vector) on the GPU or the CPU (multithreaded)? ...
Hello, I have a sparse matrix that is not symmetric I.E. the sparsity is somewhat random, and I can't count on all the values being a set distance away from the diagonal. However, it is still sparse, and I want to reduce the storage requirement on the matrix. Therefore, I am trying to figure out how to store each row starting at the f...
I have a model that already has a couple dozen of columns that will be filled most of the time. Now I need to add fields that might be different each time. what's the best approach? I don't like the EAV pattern. I don't like the idea of having a sparse table either, especially considering how these extra properties could be very differe...
Hi, I see 2 implementations of sparse matrix in this package. OpenMapRealMatrix SparseFieldMatrix Both are documented as "Sparse matrix implementation based on an open addressed map". Do you know what are the diffrences between the two? also, what's the best way to iterate over all entries in a fast way (which means, iterating over...
Hi, I need to store a 512^3 array on disk in some way and I'm currently using HDF5. Since the array is sparse a lot of disk space gets wasted. Does HDF5 provide any support for sparse array ? ...
Is there any effective implement of the solution for sparse matrix linear equation using CUDA? ...
Hi All, I need simple opinion from all Guru! I developed a program to do some matrix calculations. It work all fine with small matrix. However when I start calculating BIG thousands column row matrix. It kills the speed. I was thinking to do processing on each row and write the result in a file then free the memory and start process...
The boost ublas::compressed_matrix should only allocate space for non-zero elements. But in the below example, I am getting strange results. #include <boost/numeric/ublas/matrix_sparse.hpp> #include <boost/numeric/ublas/io.hpp> using namespace std; using namespace boost::numeric::ublas; int main () { { compressed_matrix<double,row...