*Thread Reply:* Also good idea 🙂
Gather.Town link: https://app.gather.town/app/PlXjb0deog0B4JCq/spaam-community
*Thread Reply:* so
${filepath}
is indeed a variable, that stores a line of the File_names.txt
*Thread Reply:* the while loop will read one line at a time from File_names.txt, which will modify
${filepath}
to be one line each time
*Thread Reply:* what does your file File_names.txt contain?
*Thread Reply:* is empty
*Thread Reply:* ah, that's why the while loop is not printing anything, you will need to run this before:
suffix="jpg"
find Boosted-BBB/ -type f -name "**${suffix}" > File_names.txt
*Thread Reply:* and check if the File_names.txt contains the path to your jpg files
*Thread Reply:* let me know if not, and we can continue to debug 🙂
*Thread Reply:* Thank you very much!! Do all the steps again and it worked!! To which email do I send the new script for the image sorting with new folders?
*Thread Reply:* aida_andrades@eva.mpg.de
I just saw that we have a few gather.town twins!
@Ina Wasmuth and @Sierra Blunt as blondies
And Markus and @Tre Blohm as beardy bros
*Thread Reply:* Invited you can delete this messag eif you want 🙂
*Thread Reply:* Don't apologise, I was surprised you inidicated you make that time! If you have time your morning/my afternoon today, we could quickly meet if you want?
*Thread Reply:* I just saw the message, I can any day after 3:00 pm Berlin time 😊
*Thread Reply:* Shall we book 15:00_15:45 on Monday (15th)?
*Thread Reply:* yes! Perfect!
*Thread Reply:* Hi Mohammed - depending on how you initially did your classification you may be able to push back against using MALT.
MALT is infamous for being memory intensive outside labs' computational resources, which is a large blocker for most labs. You can easily reply that to the editor and reviewer saying that.
In this case your only solution is to reduce the number of reference genomes you're inputting into your database. If you have not done it already, you could make sure you're only picking one representative per genome, for example.
However, you have to remember that MALT is not particularly special, the only benefit is that it's maybe slightly more specific because it does an alignment rather than slightly fuzzy kmer matching, and with that alignment you can generate damage plots etc. But you can also just generate alignments by mapping your self afterwards. It performs LCA just as e.g. Kraken does it, just kraken is more sensitive so you'll pick up more false positives - but you can just raise your support threshold a bit.
Furthermore both MALT 0.5.* and MALT 0.4.* are both actually broken:
http://megan.informatik.uni-tuebingen.de/t/lca-placement-failure-with-malt-v-0-5-2-and-0-5-3/1996 http://megan.informatik.uni-tuebingen.de/t/unable-to-change-the-default-coverage-parameter-for-naive-lca-assignment-with-malt-v-0-4-1/2032
This would force you to use an ancient version (0.3.8) but this will mean you will have to use a very out of date taxonomy as the version doesn't have updated Megan files anymore.
So ultimately, depending on why your reviewer wants you to run MALT you can quite robustly just say: we can't because it's broken and is outside our computational capacity (and that's with a 1TB node!)
*Thread Reply:* Hi @Mohamed Sarhan, I would also add that you can argue that MALT produces a similar taxonomic table to Kraken, especially after filtering out low abundance taxa, despite performing alignment rather than k-mer matching, by citing this paper. I used CLARK-S, which performs k-mer matching highly similar to Kraken. https://journals.asm.org/doi/full/10.1128/mSystems.00080-18
*Thread Reply:* Hi @Mohamed Sarhan, I agree with @James Fellows Yates and @irinavelsko that the lists of detected microbes are typically rather similar for Kraken and MALT providing you use similar databases (same organisms) for both. However, what Kraken absolutely cannot do is authentication, so you need some alignments for following up detected microbes. Bowtie2 alignments might be good enough for validation / authentication but they lack LCA, that is a drawback.
Now, regarding your technical issue, you can increase MALT java heap space by manually modifying the -Xmx flag in malt-build.vmoptions file which is located in /opt/malt/class folder in your malt installation. By default, I believe it is "-Xmx512m" but you can specify "-Xmx1000G" for example. Please try it and let me know whether it has worked
*Thread Reply:* Oh you can also try increasing the step size of the seeds in malt build
*Thread Reply:* I think Ron and Felix found you can reasonably go down to 8 with minimal loss of sensitivity
*Thread Reply:* Hmm, yes, I think --step 1 is the default. We tried --step 2 and --step 3, if I remember correctly, it does decrease the database size but from what we saw, it also decreases the accuracy of taxonomic classification. I do not recognize --step 8 🙂
*Thread Reply:* Thank you so much for your constructive replies. Really appreciate your help 😊 Thank you @James Fellows Yates, that's so convincing - It makes no sense to go back and use an outdated taxonomy. We used DIAMOND/MEGAN, MetaPhlAn3, and Kraken2/Bracken for taxonomic classification check, but include in the manuscript only the DIAMOND/MEGAN results. I will include these arguments and the paper @irinavelsko linked here. I hope that would be enough to convince the reviewer. Thank you @Nikolay Oskolkov - The "Xmx" was set to 64G, now changed it to 800G and it is running with the default step size 👍
*Thread Reply:* Ok - DIAMOND could be your issue there and why the reviewer is asking for MALT
*Thread Reply:* DIAMOND will not work well with short aDNA reads
*Thread Reply:* because it translates to very short amino acid sequencs and will not be specific enough
*Thread Reply:* https://peerj.com/preprints/27166/
*Thread Reply:* (which uses a BLASTX moe of MALT, but similar thing)
*Thread Reply:* Agree, this could be the reason - We will add the output of MetaPhlAn3 and Kraken2 as well. For the DIAMOND, we use it against the NCBI-nr database, that's why we like it because it gives a comprehensive picture on everything we have in our samples (Human DNA, microbiome, and dietary components). Then we keep going with further confirmation with specialized curated databases.
*Thread Reply:* Just to inform you, here is a comparison between the the bacterial assigned reads using MALT/BLASTn against the representative bacterial genomes (~16,600 genomes) and DIAMOND/BLASTx against the NCBI-nr database. The numbers of assigned reads are different from sample to sample. Looking forward to discussing this more in details during the upcoming SPAAM4 🙂
*Thread Reply:* @Mohamed Sarhan so did you manage to build the Malt DB?
*Thread Reply:* Yes, @Nikolay Oskolkov.. Thanks to your suggestion 🙏. It worked once we changed the the file you referred to. It needed ~700GB to build it.
*Thread Reply:* Good! It is interesting that MALT / BLASTN does not always assign more reads than DIAMOND / BLASTX
*Thread Reply:* Indeed... Do you have any read lengtg stats?
*Thread Reply:* Here is a read-length distribution for 4 of the samples (Just plotted them now 😄)
*Thread Reply:* Hm ok that's not what I expected
*Thread Reply:* Hmm, I would not use reads below 30 bp, but those I believe should not be assigned by DIAMOND to any organism at all because they are too short. So this does not explain why DIAMOND gives more assigned reads than MALT for some samples. Were all adapters trimmed prior to mapping with MALT?
*Thread Reply:* Yes, these are already quality-filtered adapter-trimmed merged deduplicated reads
*Thread Reply:* In my opinion it might have to do with the database itself and the sample microbial composition. I think the default word-size for the BLASTx is 6 and can be adjusted to 3 or 2 (I'm not sure about DIAMOND/BLASTx word size), but if it is so, it could mean the short-fragments of ~18 nt can be still seeded and assigned (Just in theory). What do you think?
*Thread Reply:* @Mohamed Sarhan I do not know about word size, but it seems plausible to me that this effect has to do with the NR/NT (used for Diamond) vs. 16 000 genomes (used for Malt). Since the former is much bigger, that might indeed result in more reads assigned by Diamond (higher sensitivity)
*Thread Reply:* thank you!!! ❤️