I knew I was loose, but....

I didn't know just how loose until I tried to knit a swatch for Edda. The pattern for the cardigan is a sort of modified brioche stitch (or so I'm told, I don't know precisely what brioche stitch is, but for the sweater there's knitting and purling on one side, and knitting and slipped stitches on the other side, and what you're left with is a sort of ribbing-like effect), and the recommended needle size was 7 mm. So I knit a swatch using 6.5 mm needles, knowing that I tend to knit loosely, but nope, my gauge was still too big. So I knit a second swatch using 6 mm needles, still no go. I had to drop down to 5.5 mm needles before getting the proper stitch gauge (good thing I've been collecting needles in all sizes, otherwise that would have made for many trips to the yarn store). Which seems like a big drop in needle size, no? Loose knitter indeed. I'm a little nervous now that the whole thing's going to turn out too small, even though my gauge appears to be ok. The row gauge is a little off now too, but that shouldn't be too hard to remedy.