Usual approach is to avoid using XSLT on huge documents. XSLT by design requires the whole source document to be represented as in-memory tree...
I'd try writing XmlReader-based code instead of using XSLT.
Sorry for late answer.
No, XmlReader approach assumes no XSLT is involved. XSLT requires full source document to be loaded into memory, which is you want to avoid. When using XmlReader you write code that reads XML node by node in a forward only non caching way - that's why it works great with huge XML documents, but obviously this model limits your processing - you can't pull arbitrary parts of XML - each node is visited only once.
Take a look at:
1. "Efficient Techniques for Modifying Large XML Files"
2. "Combining the XmlReader and XmlWriter classes for simple streaming transformations"