Or the small companion star is on an eccentric orbit, well off from the rotational plane of the pulsar, hence matter won't make it past the polar jets to form an accretion disc until it approaches the equatorial plane of the pulsar, then accretion can form, then the companion moves beyond that region again.
Of course, that would be easily enough found over time by modest changes in the polar jets movement, as well as rotational changes by a modest, but measurable amount.